System and Method for Providing Orientation of a Camera

ABSTRACT

A system and a method provide an orientation of a camera on a mobile device. This information can be used in an augmented reality application. The method comprises obtaining an orientation measurement from a sensor relative to the orientation of the sensor. The sensor may have different orientation compared to the camera and, thus, the orientation measurement needs to be adjusted. An appropriate transformation matrix is obtained, which maps the orientation measurement to the camera&#39;s perspective. A processor of the mobile device computes the orientation of the camera using the orientation measurement and the transformation matrix.

TECHNICAL FIELD

The following relates generally to providing orientation information ofa camera.

DESCRIPTION OF THE RELATED ART

In some augmented reality applications on mobile devices, a camera isused to capture an image of a scene and information is provided aboutthe scene. For example, an augmented reality application may show thedirection a camera is facing. The facing direction of the camera can beobtained from sensors on the mobile device, such as a magnetometer,accelerometer, or gyroscope.

BRIEF DESCRIPTION OF THE DRAWINGS

Example embodiments will now be described by way of example only withreference to the appended drawings wherein:

FIG. 1 is a schematic diagram of a mobile device and an object,according to an example embodiment.

FIG. 2 is a schematic diagram of a mobile device with rotatable partsand an object, according to an example embodiment.

FIG. 3 is a block diagram of example embodiment components of a mobiledevice configured to determine the facing direction, or orientation, orboth, of a camera.

FIG. 4 is a plan view of an example embodiment mobile device.

FIG. 5 is a plan view of another example embodiment mobile device.

FIG. 6 is a block diagram of an example embodiment of a mobile device.

FIG. 7 is a screen shot of a home screen displayed by the mobile device,according to an example embodiment.

FIG. 8 is a block diagram illustrating example embodiments of the othersoftware applications and components shown in FIG. 6.

FIG. 9 is a flow diagram of example embodiment computer executableinstructions for determining the orientation or direction, or both, of acamera based on a transformation matrix.

FIG. 10 is a flow diagram of example embodiment computer executableinstructions for obtaining data related to the configuration of a mobiledevice, and using the data to select a transformation matrix that mapsthe sensor's measurements to a camera.

FIG. 11 is a flow diagram of example embodiment computer executableinstructions for modifying a transformation matrix that maps thesensor's measurements to the camera based on dynamic inputs of themobile device's configuration.

FIG. 12 is a flow diagram of example embodiment computer executableinstructions for determining the orientation or direction, or both, oftwo cameras using transformation matrices.

DETAILED DESCRIPTION

It will be appreciated that for simplicity and clarity of illustration,where considered appropriate, reference numerals may be repeated amongthe example figures to indicate corresponding or analogous elements. Inaddition, numerous specific details are set forth in order to provide athorough understanding of the example embodiments described herein.However, it will be understood by those of ordinary skill in the artthat the example embodiments described herein may be practiced withoutthese specific details. In other instances, well-known methods,procedures and components have not been described in detail so as not toobscure the example embodiments described herein. Also, the descriptionis not to be considered as limiting the scope of the example embodimentsdescribed herein.

A mobile device can include a camera for taking video or capturing stillimages. A mobile device may also include sensors, such as amagnetometer, a gyroscope, an accelerometer, etc. for determiningorientation and direction. Some applications, such as augmented realityapplications, combine the functions of the sensors and the camera toprovide information about what the camera “sees”. For example, anapplication on the mobile device may display a picture taken by thecamera and show that, from the camera's perspective, the camera isfacing a certain direction. Such an example is shown in FIG. 1.

Turning to FIG. 1, a mobile device 100 having a camera 123 and a lightor flash device 30 is shown. The camera 123 is capturing an image of anobject 202, which is in its field of view, as represented by the dottedlines 208. A frame of reference 204 shows the directions North, South,East and West. The mobile device 100 also includes sensors 210 (e.g.magnetometer 212, gyroscope 214, accelerometer 216, etc.). An augmentedreality application shows on the display screen 12 of the mobile device100, an image of the object 202 as well as direction information. Forexample, the display 12 shows that the device 100, from the camera'sperspective, is facing North and the object 202 is located North-Westcompared to the facing direction of the camera.

It is recognized that the measurements from the sensors 210 are relativeto each of the sensors 210. The camera's sensor, the charge-coupleddevice (CCD), for example, may have a different orientation (as it ispositioned within or on the mobile device 100) from the orientation ofthe sensor (as it is positioned within or on the mobile device 100). Inother words, the measurement taken from a sensor is relative to theorientation of the sensor, and the measurement may not take intoconsideration the orientation of the camera's facing direction. Anapplication on the mobile device may be specific to that particularmobile device and the application may compute the relative direction andorientation of a camera based on the measurement by the sensor 210 (fromwhere the sensor is positioned within or on the mobile device 100).

However, different mobile devices have different configurations (e.g.different positions of sensors 210 and camera 123 within or on themobile devices) and, therefore, the relative orientation andconfiguration between the sensors 210 and the camera 123 may bedifferent across different mobile devices 100.

An application that is specific to the mobile device 100 will take intoaccount the relative positioning and orientation between the sensor 210and the camera 123, and using a transformation, use the measurementsfrom the sensor 210 to compute the direction and orientation of thecamera 123. If an application is specific to a mobile device 100, thendifferent variants of the application are used to accommodate thedifferent configurations of different mobile devices. For example, onemobile device 100 may have use one variation of an augmented realityapplication, while another mobile device 100 may use another variationof the same augmented reality application. This can become morecomplicated for developers or programmers who create augmented realityapplications because there are numerous configurations between camerasand sensors on mobile devices. Keeping track of all the differentconfigurations can be difficult, and making different variations of anapplication for the different configurations can be time consuming.

For example, continuing with FIG. 1, a magnetometer 218 having a normalvector 220 is shown. In this example, the normal vector 220 is parallelto the normal vector 209 of the camera sensor of camera 123. Theorientation of the magnetometer 218 may therefore be already physicallyaligned with the camera sensor of camera 123. Therefore, when themagnetometer 218 measures that its normal vector 220 is pointing North,the camera sensor of camera 123 is also pointing North.

In another example, in FIG. 1, the magnetometer 222 may be in adifferent configuration and its normal vector 224 may be pointing up. Inother words, the normal 224 of the magnetometer 222 does not align withthe normal vector 209 of the camera 123. In such an example, ameasurement from the magnetometer 222 indicating that it is pointingNorth, does not mean that the camera sensor of camera 123 is pointingNorth. A computation is required to translate or map the measurementfrom the magnetometer 222 to the orientation of the camera 123. It canbe appreciated that there are various other example configurationsbetween other sensors (e.g. gyroscope, accelerometer, etc.) and othercameras. For example, there can be a forward facing camera and abackward facing camera. It can be appreciated that the mappings can bedifferent between different sensors and cameras.

It can be appreciated that the orientation and position of a sensor 210can affect the meaning or interpretation of the measurement data outputby the sensor 210.

The proposed systems and methods described herein take into account thevarious configurations and provide a database of transformation matricesused to map measurements from a sensor to a camera. Methods are alsoprovided for determining which transformation matrix is to be used, andif it is to be modified.

Turning to FIG. 2, another example configuration of a mobile device 100is shown. The mobile device 100 is considered a flip phone which has afirst part 228 and a second part 226 that are able to move relative toeach other. For example, they can rotate up and down relative to eachother. In another example embodiment, the first part 228 and the secondpart 226 can swivel or rotate in various different axes. For example,there may be a multi-axis joint 230 connecting the first part 228 andthe second part 226. The joint 230 may, for example, be a ball joint.

The mobile device 100 may also include one or more auxiliary sensors 240to measure the relative positioning of the first part 228 and the secondpart 226. For example, the auxiliary sensor 240 is able to determinewhether the mobile device 100 is flipped open, closed, in anintermediate opened position, in a swivelled position, the angle ofrotation along various axes, etc. Non-limiting examples of the auxiliarysensor or sensors 240 include a transducer, a photo resistor, a switch,an optical switch, a proximity sensor, a magnetic sensor, andcombinations thereof. The relative positioning of the parts 228, 226 canalso be measured or computed by taking the measurements from thegyroscope 214 or accelerometer 216, or both. It can be appreciated thatother sensors suitable for measuring the relative positioning andorientation of the various parts of the mobile device 100 can also beused.

The camera 123 may be mounted or positioned within the first part 228 ofthe mobile device 100, while the sensor or sensors 210 may be positionedin the second part 226. As the parts 228, 226 move relative to eachother, the sensor or sensors 210 also move relative to the camera 123.In one example embodiment, if a sensor 232 (e.g. magnetometer,gyroscope, accelerometer, etc.) is positioned in the second part 226,which is at an angle to the first part 228, the sensor's 232 normalvector 236 is pointed at a downward angle relative to the camera'snormal vector 209.

In another example embodiment in FIG. 2, the sensor 234 is located inthe second part 226, and its normal vector 238 is pointed at an upwardangle relative to the normal vector 209 of the camera 123.

The mapping of the direction or orientation measurements from thesensors can, therefore, change relative the camera 123. The mapping ortransformation of the data from the sensors 232, 234, 210 (e.g.magnetometers, accelerometers, gyroscopes, etc.) to the camera 123depends on their relative positioning and orientation, which can changein the flip phone, as shown, for example, in FIG. 2. The mappingfunction or transformation process (e.g. the transformation matrix) maytake into account the measurement data from the auxiliary sensor orsensors 240 to map or transform the direction or orientationmeasurements outputted from the sensor or sensors 210 to the directionor orientation of the camera 123. The transformation process ortransformation matrix can be considered dynamic and may depend on thedata provided by the auxiliary sensor or sensors 240. The parametersused in the transformation process may be represented as atransformation matrix.

In an example embodiment, a transformation matrix contains values thatcan be used to transform the measurement data of a sensor, which is fromthe perspective of the sensor package, to measurement data that is fromthe perspective of a camera. For example, the values in thetransformation matrix can include rotation angles to adjust one or moreaxes, and translations of distance to adjust for the differentpositioning between a sensor and a camera.

The example embodiments described herein also apply to other mobiledevices having parts that move relative to one another. Non-limitingexamples include mobile devices that have one part slide relative toanother part, and laptops or other mobile devices that have a displayportion able to flip up and down as well as swivel around relative to abase portion.

Turning to FIG. 3, an example configuration of a mobile device 100 isshown. The mobile device 100 may include one or more cameras 244, 245.For example, a camera may have one camera, two cameras, or any othernumber of cameras. The cameras can be controlled and monitored by acamera application 246. The camera application 246 can determine whichcamera or cameras are being activated. The camera application 246 is incommunication with an application 248 for determining the facingdirection or orientation, or both, of a given camera on the mobiledevice 100.

The mobile device 100 also includes one or more direction or orientationsensors 210, or both (e.g. magnetometer, gyroscope, accelerometer,etc.). Such a sensor or sensors are in communication with theapplication 248 for determining the facing direction or orientation, orboth, of a given camera on the mobile device 100. Similarly, theauxiliary sensor or sensors 240 for determining the relative position ororientation, or both, of a camera and a sensor are also in communicationwith the application 240.

Continuing with FIG. 3, the mobile device 100 includes a transformationmatrix database 254 which stores one or more transformation matrices.Each transformation matrix can be applied to the data output of a givensensor 210 to compute the direction or orientation of a given camera242. Data is associated with each transformation matrix to helpdetermine and select which transformation matrix is most appropriate fora given situation.

For example, the data associated with each transformation matrix in thedatabase 254 may include a model of the mobile device, a cameraidentity, a sensor identity and a dynamic input (e.g. from one or moreauxiliary sensors 240). It can be appreciated that other data may beassociated with a transformation matrix.

It can also be appreciated that the model of the mobile device is a wayto identify the configuration of a mobile device 100. Different modelsof mobile devices may have different camera and sensor configurations,and thus, a particular model of a mobile device is associated with aparticular transformation matrix.

The model of the mobile device, the camera identity, and the sensoridentity can be, for example, represented with numbers, letters, orcombinations thereof.

The application 248 for determining the facing direction or orientation,or both, of a camera is in communication with the database 248.

In an example embodiment, the application 248 obtains the cameraidentity of which camera is being activated, for example, from thecamera application 246. The application 248 also obtains the measurementdata from a sensor 210, as well as the identity of the sensor providingthe measurement data. Depending on the type or model of the mobiledevice, may also obtain dynamic inputs from one or more auxiliarysensors 240. The application 248 also obtains the model of the mobiledevice. For example, the application 248 may obtain the model from themobile device's system parameters. The application 248 may use the modelof the mobile device to select an appropriate transformation matrix fromthe database 254.

In models of mobile devices that have multiple orientation or directionsensors 210, or both, the application 248 may also use the sensoridentity to search the database 254 for an appropriate transformationmatrix. Similarly, if the given model of a mobile device has multiplecameras, the application 248 may use the camera identity to search thedatabase 254 for an appropriate transformation matrix. In other words,the application 248 can use different types of information (e.g. modelof the device, camera identity, sensor identity, or combinationsthereof) to search for the appropriate transformation matrix. Afterdetermining the appropriate transformation matrix, it may modify thetransformation matrix according to the dynamic inputs, for example, asprovided by the auxiliary sensor or sensors 240.

The application 248 may then use the obtained transformation matrix andthe measurement data of the orientation or direction sensor 210 tocompute the orientation or direction of the camera 242. This orientationor direction of the camera may be provided to an augmented realityapplication 252.

It can be appreciated that various mobile devices can be used with theexample embodiments described herein. Examples of applicable electronicdevices include pagers, tablets, cellular phones, cellular smart-phones,wireless organizers, personal digital assistants, computers, laptops,handheld wireless communication devices, wirelessly enabled notebookcomputers, camera devices and the like. Such devices will hereinafter becommonly referred to as “mobile devices” for the sake of clarity. Itwill however be appreciated that the example embodiments describedherein are also suitable for other devices, e.g. “non-mobile” devices.

In an example embodiment, the mobile device is a two-way communicationdevice with advanced data communication capabilities including thecapability to communicate with other mobile devices or computer systemsthrough a network of transceiver stations. The mobile device may alsohave the capability to allow voice communication. Depending on thefunctionality provided by the mobile device, it may be referred to as adata messaging device, a two-way pager, a cellular telephone with datamessaging capabilities, a wireless Internet appliance, or a datacommunication device (with or without telephony capabilities).

Referring to FIGS. 4 and 5, an example embodiment of a mobile device 100a is shown in FIG. 4, and another example embodiment of a mobile device100 b is shown in FIG. 5. It will be appreciated that the numeral “100”will hereinafter refer to any mobile device 100, including the exampleembodiments 100 a and 100 b, those example embodiments enumerated aboveor otherwise. It will also be appreciated that a similar numberingconvention may be used for other general features common between allFigures such as a display 12, a positioning device 14, a cancel orescape button 16, a camera button 17, and a menu or option button 24.

The mobile device 100 a shown in FIG. 4 comprises a display 12 a and thecursor or view positioning device 14 shown in this example embodiment isa trackball 14 a. Positioning device 14 may serve as another inputmember and is both rotational to provide selection inputs to the mainprocessor 102 (see FIG. 5) and can also be pressed in a directiongenerally toward housing to provide another selection input to theprocessor 102. Trackball 14 a permits multi-directional positioning ofthe selection cursor 18 (see FIG. 7) such that the selection cursor 18can be moved in an upward direction, in a downward direction and, ifdesired and/or permitted, in any diagonal direction. The trackball 14 ais in this example situated on the front face of a housing for mobiledevice 100 a as shown in FIG. 4 to enable a user to manoeuvre thetrackball 14 a while holding the mobile device 100 a in one hand. Thetrackball 14 a may serve as another input member (in addition to adirectional or positioning member) to provide selection inputs to theprocessor 102 and can preferably be pressed in a direction towards thehousing of the mobile device 100 b to provide such a selection input.

The display 12 may include a selection cursor 18 that depicts generallywhere the next input or selection will be received. The selection cursor18 may comprise a box, alteration of an icon or any combination offeatures that enable the user to identify the currently chosen icon oritem. The mobile device 100 a in FIG. 4 also comprises a programmableconvenience button 15 to activate a selected application such as, forexample, a calendar or calculator. Further, mobile device 100 a includesan escape or cancel button 16 a, a camera button 17 a, a menu or optionbutton 24 a and a keyboard 20. The camera button 17 is able to activatephoto-capturing functions when pressed preferably in the directiontowards the housing. The menu or option button 24 loads a menu or listof options on display 12 a when pressed. In this example, the escape orcancel button 16 a, the menu option button 24 a, and keyboard 20 aredisposed on the front face of the mobile device housing, while theconvenience button 15 and camera button 17 a are disposed at the side ofthe housing. This button placement enables a user to operate thesebuttons while holding the mobile device 100 in one hand. The keyboard 20is, in this example embodiment, a standard QWERTY keyboard.

The mobile device 100 b shown in FIG. 5 comprises a display 12 b and thepositioning device 14 in this example embodiment is a trackball 14 b.The mobile device 100 b also comprises a menu or option button 24 b, acancel or escape button 16 b, and a camera button 17 b. The mobiledevice 100 b as illustrated in FIG. 5, comprises a reduced QWERTYkeyboard 22. In this example embodiment, the keyboard 22, positioningdevice 14 b, escape button 16 b and menu button 24 b are disposed on afront face of a mobile device housing. The reduced QWERTY keyboard 22comprises a plurality of multi-functional keys and corresponding indiciaincluding keys associated with alphabetic characters corresponding to aQWERTY array of letters A to Z and an overlaid numeric phone keyarrangement.

It will be appreciated that for the mobile device 100, a wide range ofone or more positioning or cursor/view positioning mechanisms such as atouch pad, a positioning wheel, a joystick button, a mouse, atouchscreen, a set of arrow keys, a tablet, an accelerometer (forsensing orientation and/or movements of the mobile device 100 etc.), orother whether presently known or unknown may be employed. Similarly, anyvariation of keyboard 20, 22 may be used. It will also be appreciatedthat the mobile devices 100 shown in FIGS. 4 and 5 are for illustrativepurposes only and various other mobile devices 100 are equallyapplicable to the following examples. For example, other mobile devices100 may include the trackball 14 b, escape button 16 b and menu oroption button 24 similar to that shown in FIG. 5 only with a full orstandard keyboard of any type. Other buttons may also be disposed on themobile device housing such as colour coded “Answer” and “Ignore” buttonsto be used in telephonic communications. In another example, the display12 may itself be touch sensitive thus itself providing an inputmechanism in addition to display capabilities.

To aid the reader in understanding the structure of the mobile device100, reference will now be made to FIGS. 6 through 8.

Referring first to FIG. 6, shown therein is a block diagram of anexample embodiment of a mobile device 100. The mobile device 100comprises a number of components such as a main processor 102 thatcontrols the overall operation of the mobile device 100. Communicationfunctions, including data and voice communications, are performedthrough a communication subsystem 104. The communication subsystem 104receives messages from and sends messages to a wireless network 200. Inthis example embodiment of the mobile device 100, the communicationsubsystem 104 is configured in accordance with the Global System forMobile Communication (GSM) and General Packet Radio Services (GPRS)standards, which is used worldwide. Other communication configurationsthat are equally applicable are the 3G and 4G networks such as EDGE,UMTS and HSDPA, LTE, Wi-Max etc. New standards are still being defined,but it is believed that they will have similarities to the networkbehaviour described herein, and it will also be understood by personsskilled in the art that the example embodiments described herein areintended to use any other suitable standards that are developed in thefuture. The wireless link connecting the communication subsystem 104with the wireless network 200 represents one or more different RadioFrequency (RF) channels, operating according to defined protocolsspecified for GSM/GPRS communications.

The main processor 102 also interacts with additional subsystems such asa Random Access Memory (RAM) 106, a flash memory 108, a display 110, anauxiliary input/output (I/O) subsystem 112, a data port 114, a keyboard116, a speaker 118, a microphone 120, a GPS receiver 121, short-rangecommunications 122, a camera 123, a camera light or flash 30, and otherdevice subsystems 124.

The main processor 102 may also interact with sensors 210 and auxiliarysensors 240. As shown in FIG. 6, the main processor 102 is interactingwith a magnetometer 212, a gyroscope 214, and an accelerometer 216.

Some of the subsystems of the mobile device 100 performcommunication-related functions, whereas other subsystems may provide“resident” or on-device functions. By way of example, the display 110and the keyboard 116 may be used for both communication-relatedfunctions, such as entering a text message for transmission over thenetwork 200, and device-resident functions such as a calculator or tasklist.

The mobile device 100 can send and receive communication signals overthe wireless network 200 after required network registration oractivation procedures have been completed. Network access is associatedwith a subscriber or user of the mobile device 100. To identify asubscriber, the mobile device 100 may use a subscriber module componentor “smart card” 126, such as a Subscriber Identity Module (SIM), aRemovable User Identity Module (RUIM) and a Universal SubscriberIdentity Module (USIM). In the example shown, a SIM/RUIM/USIM 126 is tobe inserted into a SIM/RUIM/USIM interface 128 in order to communicatewith a network. Without the component 126, the mobile device 100 is notfully operational for communication with the wireless network 200. Oncethe SIM/RUIM/USIM 126 is inserted into the SIM/RUIM/USIM interface 128,it is coupled to the main processor 102.

The mobile device 100 is a battery-powered device and includes a batteryinterface 132 for receiving one or more rechargeable batteries 130. Inat least some example embodiments, the battery 130 can be a smartbattery with an embedded microprocessor. The battery interface 132 iscoupled to a regulator (not shown), which assists the battery 130 inproviding power V+ to the mobile device 100. Although current technologymakes use of a battery, future technologies such as micro fuel cells mayprovide the power to the mobile device 100.

The mobile device 100 also includes an operating system 134 and softwarecomponents 136 to 146 which are described in more detail below. Theoperating system 134 and the software components 136 to 146 that areexecuted by the main processor 102 are typically stored in a persistentstore such as the flash memory 108, which may alternatively be aread-only memory (ROM) or similar storage element (not shown). Thoseskilled in the art will appreciate that portions of the operating system134 and the software components 136 to 146, such as specific deviceapplications, or parts thereof, may be temporarily loaded into avolatile store such as the RAM 106. Other software components can alsobe included, as is well known to those skilled in the art.

The subset of software applications 136 that control basic deviceoperations, including data and voice communication applications, may beinstalled on the mobile device 100 during its manufacture. Softwareapplications may include a message application 138, a device statemodule 140, a Personal Information Manager (PIM) 142, a connect module144 and an IT policy module 146. A message application 138 can be anysuitable software program that allows a user of the mobile device 100 tosend and receive electronic messages, wherein messages are typicallystored in the flash memory 108 of the mobile device 100. A device statemodule 140 provides persistence, i.e. the device state module 140ensures that important device data is stored in persistent memory, suchas the flash memory 108, so that the data is not lost when the mobiledevice 100 is turned off or loses power. A PIM 142 includesfunctionality for organizing and managing data items of interest to theuser, such as, but not limited to, e-mail, contacts, calendar events,and voice mails, and may interact with the wireless network 200. Aconnect module 144 implements the communication protocols that arerequired for the mobile device 100 to communicate with the wirelessinfrastructure and any host system, such as an enterprise system, thatthe mobile device 100 is authorized to interface with. An IT policymodule 146 receives IT policy data that encodes the IT policy, and maybe responsible for organizing and securing rules such as the “SetMaximum Password Attempts” IT policy.

Other types of software applications or components 139 can also beinstalled on the mobile device 100. These software applications 139 canbe pre-installed applications (i.e. other than message application 138)or third party applications, which are added after the manufacture ofthe mobile device 100. Examples of third party applications includegames, calculators, utilities, etc.

The additional applications 139 can be loaded onto the mobile device 100through at least one of the wireless network 200, the auxiliary I/Osubsystem 112, the data port 114, the short-range communicationssubsystem 122, or any other suitable device subsystem 124.

The data port 114 can be any suitable port that enables datacommunication between the mobile device 100 and another computingdevice. The data port 114 can be a serial or a parallel port. In someinstances, the data port 114 can be a USB port that includes data linesfor data transfer and a supply line that can provide a charging currentto charge the battery 130 of the mobile device 100.

For voice communications, received signals are output to the speaker118, and signals for transmission are generated by the microphone 120.Although voice or audio signal output is accomplished primarily throughthe speaker 118, the display 110 can also be used to provide additionalinformation such as the identity of a calling party, duration of a voicecall, or other voice call related information.

Turning now to FIG. 7, the mobile device 100 may display a home screen40, which can be set as the active screen when the mobile device 100 ispowered up and may constitute the main ribbon application. The homescreen 40 generally comprises a status region 44 and a theme background46, which provides a graphical background for the display 12. The themebackground 46 displays a series of icons 42 in a predefined arrangementon a graphical background. In some themes, the home screen 40 may limitthe number icons 42 shown on the home screen 40 so as to not detractfrom the theme background 46, particularly where the background 46 ischosen for aesthetic reasons. The theme background 46 shown in FIG. 7provides a grid of icons. It will be appreciated that preferably severalthemes are available for the user to select and that any applicablearrangement may be used. An example icon may be an augmented realityicon 51 used to indicate an augmented reality application, for example,for determining direction or orientation. One or more of the series oficons 42 is typically a folder 52 that itself is capable of organizingany number of applications therewithin.

The status region 44 in this example embodiment comprises a date/timedisplay 48. The theme background 46, in addition to a graphicalbackground and the series of icons 42, also comprises a status bar 50.The status bar 50 provides information to the user based on the locationof the selection cursor 18, e.g. by displaying a name for the icon 53that is currently highlighted.

An application, such as message application 138 may be initiated (openedor viewed) from display 12 by highlighting a corresponding icon 53 usingthe positioning device 14 and providing a suitable user input to themobile device 100. For example, message application 138 may be initiatedby moving the positioning device 14 such that the icon 53 is highlightedby the selection box 18 as shown in FIG. 7, and providing a selectioninput, e.g. by pressing the trackball 14 b.

FIG. 8 shows an example of the other software applications andcomponents 139 that may be stored and used on the mobile device 100.Only examples are shown in FIG. 8 and such examples are not to beconsidered exhaustive. In this example, an alarm application 54 may beused to activate an alarm at a time and date determined by the user. AGPS application 56 may be used to determine the location of a mobiledevice. A calendar application 58 that may be used to organizeappointments. Another example application is an augmented realityapplication 252 that may be used to take video or camera images andprovide additional information (e.g. direction or orientation data) withthe video or camera images. There may also be an application 248 fordetermining the facing direction or orientation, or both, of a camera.Another application shown is an address book 62 that is used to storecontact information which may include, for example, a phone number, nameand e-mail address.

It will be appreciated that any module or component exemplified hereinthat executes instructions or operations may include or otherwise haveaccess to computer readable media such as storage media, computerstorage media, or data storage devices (removable and/or non-removable)such as, for example, magnetic disks, optical disks, or tape. Computerstorage media may include volatile and non-volatile, removable andnon-removable media implemented in any method or technology for storageof information, such as computer readable instructions, data structures,program modules, or other data, except transitory propagating signalsper se. Examples of computer storage media include RAM, ROM, EEPROM,flash memory or other memory technology, CD-ROM, digital versatile disks(DVD) or other optical storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices, or any othermedium which can be used to store the desired information and which canbe accessed by an application, module, or both. Any such computerstorage media may be part of the mobile device 100, or accessible orconnectable thereto. Any application or module herein described may beimplemented using computer readable/executable instructions oroperations that may be stored or otherwise held by such computerreadable media.

Referring to FIG. 9, example computer executable or processorimplemented instructions are provided for providing orientation ordirection data for a camera on a mobile device 100. At block 260, themobile device 100 obtains the orientation or direction measurement froma sensor 210. The measurement data is relative to the sensor'sorientation. The direction, for example, can be a North, South, East,West type direction which can be measured by a magnetometer. Orientationmeasurements can be, for example, forward, backward, left, right, up anddown, clockwise, or counter clockwise. Such measurements can, forexample, be measured by a gyroscope or accelerometer, or both.

The mobile device 100, at block 262, determines a transformation matrixthat maps the sensor's measurement data to a camera 123 on the mobiledevice. The transformation matrix, for example, can be selected from adatabase 254, or can be selected from the database 254 and thenmodified, or can be computed. It can be appreciated that there aredifferent approaches to determining the transformation matrix.

At block 264, the mobile device 100 applies the transformation matrix tothe sensor's measurement data to compute the orientation or direction,or both, of the camera. In an example embodiment, the orientation whichthe CCD of the camera faces is computed.

Turning to FIG. 10, example computer executable or processor implementedinstructions are provided for selecting a transformation matrix. Atblock 266, the mobile device 100 obtains orientation or directionmeasurements from a sensor relative to the sensor's orientation. Atblock 268, the mobile device 100 obtains data related to theconfiguration of the mobile device 100. Non-limiting examples ofobtaining such data include: obtaining the model of the mobile device(block 274); obtaining the identity or identification of the sensorproviding the measurement data (block 276); obtaining the identity oridentification of a given camera, or given cameras, that are beingactivated (block 278); and obtaining the dynamic inputs of the mobiledevice's configuration (e.g. swivel or rotation of a flip phone, slidingparts, etc.) (block 280).

Continuing with FIG. 10, at block 270, the mobile device 100 uses theobtained data to select from a database 254 of transformation matrices atransformation matrix that maps the sensor's measurement data to a givencamera on the mobile device 100. For example, if the obtained datamatches the data in the database 254, the transformation matrixassociated with the matching data in the database is selected.

The obtained transformation matrix and the sensor's measurement data areused to compute the orientation or direction, or both, of the camera(block 272). For example, the sensor's measurement data may be presentedin the form of a matrix, and the measurement data and the transformationmatrix are multiplied (e.g. using matrix multiplication) to compute theorientation or direction data, or both, of a camera. Other methods forapplying the transformation matrix are also applicable.

Turning to FIG. 11, example computer executable or processor implementedinstructions are provided for computing a transformation matrix, takinginto account dynamic inputs, for example from one or more auxiliarysensors 240. At block 282, the mobile device 100 obtains the orientationor direction measurement from a sensor relative to the sensor'sorientation. At block 284, the mobile device 100 obtains theidentification of the sensors providing the measurement. The model ofthe mobile device is obtained (block 286) and the identification of thecamera or cameras that are being activated are obtained (block 288). Themodel of the mobile device 100 is used to select from the database 254 atransformation matrix that maps the given sensor's measurement data to acamera on the mobile device. The identification of the camera and theidentification of the sensor may also be used to search the database254, for example, where there are multiple cameras and multiple sensors210 within a given model of a mobile device.

Continuing with FIG. 11, the dynamic inputs relating to the mobiledevice's configuration are obtained (block 292). The dynamic inputs maydetermine, for example, whether a flip phone is flipped open, whether asliding portion is slid to an open position, whether a first part of amobile device is oriented at some angle relative to a second part of themobile device, etc. The dynamic inputs may represent angles or positionsand these dynamic inputs are used to modify the selected transformationmatrix (block 294). In an example embodiment, there may be one or morevariables in the selected transformation matrix that are initiallyunknown when obtained from the database 254. After obtaining the dynamicinputs from the auxiliary sensor or sensors 240, the dynamic inputs areused to compute the values of variables in the selected transformationmatrix. In another example embodiment, the value(s) of the one or morevariables are the dynamic inputs.

In another example embodiment, the transformation matrix is modified byperforming an operation on the transformation matrix that includes thevalues of the dynamic inputs. For example, the dynamic inputs can berepresented in the form of a matrix. The dynamic inputs can then beadded or multiplied with the transformation matrix to compute a modifiedtransformation matrix.

At block 296, the modified transformation matrix is applied to thesensor's measurement data to compute the orientation or direction, orboth, of the camera. It can be appreciated that if the dynamic inputchanges value, then the operations at blocks 294 and 296 are re-computedto determine a new modified transformation matrix and to determine thenew orientation or direction of the camera. Similarly, if other new datais provided (e.g. sensor's measurement data, different sensor beingused, different camera being used, etc.), then the camera's orientationor direction may be re-computed. The updated computation may or may notinvolve selecting a new transformation matrix. For example, if adifferent sensor or camera on the mobile device are being used, then anew transformation matrix may be selected.

Turning to FIG. 12, example computer executable or processor implementedinstructions are also provided for computing the direction ororientation of two or more cameras on a mobile device 100. This may beused when a mobile device 100 has multiple cameras. For example, amobile device may have a front facing camera (e.g. looking away from thefront of the user to give the user's perspective) and a back facingcamera (e.g. looking towards the user to capture images of the user orscenes behind the user). The facing directions or orientations of two ormore cameras can be computed. In another example, an augmented realityapplication may activate two or more cameras and show on the display 12the images or video captured by the two or more cameras. Otherinformation, such as the direction or orientation of each camera may bealso be shown on the display 12.

At block 298, the mobile device 100 obtains orientation or directionmeasurement data from a sensor. The measurement data is, for example,relative to the sensor's orientation. At block 300, the mobile device100 obtains the identification of the sensor providing the measurement.At block 302, the mobile device 100 obtains the model of the mobiledevice (e.g. itself). At block 304, the mobile device 100 obtains theidentification of which cameras are being activated. For example, it mayobtain an identification for each of the two or more cameras that arebeing activated.

In the example provided in FIG. 12, one camera is referred to as cameraA, and another camera is referred to as camera B. With respect to cameraA, at block 306, the mobile device 100 uses the obtained data to selectfrom a database 254 of transformation matrices a transformation matrixthat maps the sensor's measurement data to camera A on the mobile device100. At block 308, dynamic inputs describing the mobile device'sconfiguration are obtained. For example, the dynamic inputs are based onthe measurements from the auxiliary sensors 240. After obtaining thedynamic inputs and selecting an appropriate transformation matrix, themobile device 100 modifies the transformation matrix to account for thedynamic inputs (block 310). At block 312, the modified transformationmatrix and the sensor's measurement data are used to compute theorientation or direction, or both, of camera A.

Similar to blocks 306, 308, 310 and 312, for camera B, the mobile device100 selects an appropriate transformation matrix from the database 254(block 314), obtains dynamic inputs describing the mobile device'sconfiguration (block 316), modifies the selected transformation matrixaccording to the dynamic inputs (block 318), and applies the modifiedtransformation matrix to the sensor's measurement data to compute theorientation or direction, or both, of camera B (block 320). It can beappreciated that although the general process may be similar forcomputing the orientation or direction, or both, of the cameras, theselected transformation matrices may be different because the relativeorientation and positioning between each camera and a sensor may bedifferent. Similarly, how the dynamic inputs are used to modify theselected transformation matrix may also be different for each camera.Therefore, it can be appreciated that although the same direction ororientation measurement data from a sensor 210 is used for each camera,the computed direction or orientation of each camera may be different.

The example embodiment systems and methods described herein makes iteasier for a software developer to create applications involving the useof a sensor 210 and a camera 123, for example in augmented realityapplications. In particular, a developer does not need to create customapplications for each model of a mobile device to take into account theparticular configuration between a sensor 210 and a camera 123. Usingthe example embodiments described herein, a transformation matrix isselected, and in some cases is modified, to compute the orientation ordirection of a camera. The transformation matrix is appropriate for themodel of the mobile device. In other words, the developer does not needto consider the numerous models of mobile devices that are availablebecause the systems and methods described herein will account for suchnumerous variations.

The example embodiments described herein also allow more flexibility todevelop applications for mobile devices with moving parts (e.g. flipphones, sliding bars, swivel tops, etc.). The example embodimentsdescribed herein can take into account the different configurations of amobile device 100.

In general, a method is provided for providing an orientation of acamera on a mobile device. The method includes: obtaining an orientationmeasurement from a sensor relative to the orientation of the sensor;determining a transformation matrix that maps the orientationmeasurement to the camera; and computing the orientation of the camerausing the orientation measurement and the transformation matrix.

In another example aspect, the transformation matrix includes at leastone value that changes the orientation of the orientation measurement.In another example aspect, the sensor is at least one of a magnetometer,a gyroscope and an accelerometer. In another example aspect, the methodfurther includes obtaining data related to the configuration of themobile device. In another example aspect, the method further includesusing the obtained data to select from a database of transformationmatrices the transformation matrix. In another example aspect, obtainingthe data includes obtaining a model identification of a mobile device.In another example aspect, obtaining the data includes obtaining anidentification of the sensor and an identification of the camera. Inanother example aspect, obtaining the data includes obtaining dynamicinputs describing a configuration of the mobile device, the dynamicinputs based on data provided by an auxiliary sensor on the mobiledevice. In another example aspect, the mobile device includes at leasttwo parts able to move relative to each other, and the method furtherincludes the auxiliary sensor providing the data describing theconfiguration of the at least two parts of the mobile device. In anotherexample aspect, the mobile device is a flip phone. In another exampleaspect, the method further includes determining if the camera is beingactivated, and if activated, determining the transformation matrix.

In general, a mobile device is provided for providing the orientation ofa camera. The mobile device includes: the camera; a sensor configured toprovide an orientation measurement relative to the orientation of thesensor; and a processor configured to determine a transformation matrixthat maps the orientation measurement to the camera and compute theorientation of the camera using the orientation measurement and thetransformation matrix.

In general, a method is provided for providing an orientation of acamera on a mobile device. The method includes: obtaining an orientationmeasurement from a sensor relative to the orientation of the sensor;obtaining a model of the mobile device; obtaining an identification ofthe camera that is currently activated; searching a database oftransformation matrices using at least the model of the mobile deviceand the identification of the camera as search parameters to select atransformation matrix; and applying the transformation matrix to theorientation measurement to compute the orientation of the camera.

The steps or operations in the flow charts described herein are just forexample. There may be many variations to these steps or operationswithout departing from the spirit of the invention or inventions. Forinstance, the steps may be performed in a differing order, or steps maybe added, deleted, or modified.

It will be appreciated that the particular example embodiments shown inthe figures and described above are for illustrative purposes only andmany other variations can be used according to the example embodimentsdescribed. Although the above has been described with reference tocertain specific example embodiments, various modifications thereof willbe apparent to those skilled in the art as outlined in the appendedclaims.

1. A method implemented by a processor, of providing an orientation of acamera on a mobile device, comprising: obtaining an orientationmeasurement from a sensor relative to the orientation of the sensor;determining a transformation matrix that maps the orientationmeasurement to the camera; and computing the orientation of the camerausing the orientation measurement and the transformation matrix.
 2. Themethod of claim 1 wherein the transformation matrix includes at leastone value that changes the orientation of the orientation measurement.3. The method of claim 1 wherein the sensor is at least one of amagnetometer, a gyroscope and an accelerometer.
 4. The method of claim 1further comprising obtaining data related to the configuration of themobile device.
 5. The method of claim 4 further comprising using theobtained data to select from a database of transformation matrices thetransformation matrix.
 6. The method of claim 4 wherein obtaining thedata comprises obtaining a model identification of a mobile device. 7.The method of claim 4 wherein obtaining the data comprises obtaining anidentification of the sensor and an identification of the camera.
 8. Themethod of claim 4 wherein obtaining the data comprises obtaining dynamicinputs describing a configuration of the mobile device, the dynamicinputs based on data provided by an auxiliary sensor on the mobiledevice.
 9. The method of claim 8 wherein the mobile device comprises atleast two parts able to move relative to each other, and the methodfurther comprising the auxiliary sensor providing the data describingthe configuration of the at least two parts of the mobile device. 10.The method of claim 9 wherein the mobile device is a flip phone.
 11. Themethod of claim 1 further comprising determining if the camera is beingactivated, and if activated, determining the transformation matrix. 12.A mobile device for providing the orientation of a camera, the mobiledevice comprising: the camera; a sensor configured to provide anorientation measurement relative to the orientation of the sensor; and aprocessor configured to determine a transformation matrix that maps theorientation measurement to the camera and compute the orientation of thecamera using the orientation measurement and the transformation matrix.13. The mobile device of claim 12 wherein the transformation matrixincludes at least one value that changes the orientation of theorientation measurement.
 14. The mobile device of claim 12 wherein thesensor is at least one of a magnetometer, a gyroscope and anaccelerometer.
 15. The mobile device of claim 12 wherein the processoris further configured to obtain data related to the configuration of themobile device.
 16. The mobile device of claim 15 wherein the processoris further configured to use the obtained data to select from a databaseof transformation matrices the transformation matrix.
 17. The mobiledevice of claim 15 wherein the processor is configured to obtain thedata by obtaining a model identification of a mobile device.
 18. Themobile device of claim 15 wherein the processor is configured to obtainthe data by obtaining an identification of the sensor and anidentification of the camera.
 19. The mobile device of claim 15 furthercomprising an auxiliary sensor and wherein the processor is configuredto obtain the data by obtaining dynamic inputs describing aconfiguration of the mobile device, the dynamic inputs based on dataprovided by the auxiliary sensor.
 20. The mobile device of claim 19wherein the mobile device comprises at least two parts able to moverelative to each other, and the auxiliary sensor is configured toprovide the data describing the configuration of the at least two partsof the mobile device.
 21. The mobile device of claim 20 wherein themobile device is a flip phone.
 22. The mobile device of claim 12 whereinthe processor is further configured to determine if the camera isactivated, and if activated, determine the transformation matrix.
 23. Amethod implemented by a processor, of providing an orientation of acamera on a mobile device, comprising: obtaining an orientationmeasurement from a sensor relative to the orientation of the sensor;obtaining a model of the mobile device; obtaining an identification ofthe camera that is currently activated; searching a database oftransformation matrices using at least the model of the mobile deviceand the identification of the camera as search parameters to select atransformation matrix; and applying the transformation matrix to theorientation measurement to compute the orientation of the camera.